{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Naive Bayes and Logistic Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this tutorial, we'll explore training and evaluation of Naive Bayes and Logitistic Regression Classifiers.\n",
    "\n",
    "To start, we import the standard BIDMach class definitions. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 CUDA device found, CUDA version 8.0\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "\u001b[32mimport \u001b[39m\u001b[36m$exec.$                          \u001b[39m"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import $exec.^.lib.bidmach_notebook_init"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we load some training and test data, and some category labels. The data come from a news collection from Reuters, and is a \"classic\" test set for classification. Each article belongs to one or more of 103 categories. The articles are represented as Bag-of-Words (BoW) column vectors. For a data matrix A, element A(i,j) holds the count of word i in document j. \n",
    "\n",
    "The category matrices have 103 rows, and a category matrix C has a one in position C(i,j) if document j is tagged with category i, or zero otherwise.  \n",
    "\n",
    "To reduce the computing time and memory footprint, the training data have been sampled. The full collection has about 700k documents. Our training set has 60k. \n",
    "\n",
    "Since the document matrices contain counts of words, we use a min function to limit the count to \"1\", i.e. because we need binary features for naive Bayes. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[36mdict\u001b[39m: \u001b[32mString\u001b[39m = \u001b[32m\"../data/rcv1/\"\u001b[39m\n",
       "\u001b[36mtraindata\u001b[39m: \u001b[32mSMat\u001b[39m = (    33,     0)     1\n",
       "(    47,     0)     1\n",
       "(    94,     0)     1\n",
       "(   104,     0)     1\n",
       "(   112,     0)     1\n",
       "(   118,     0)     1\n",
       "(   141,     0)     1\n",
       "(   165,     0)     1\n",
       "(   179,     0)     1\n",
       "(   251,     0)     1\n",
       "(   270,     0)     1\n",
       "(   306,     0)     1\n",
       "\u001b[33m...\u001b[39m\n",
       "\u001b[36mtraincats\u001b[39m: \u001b[32mFMat\u001b[39m =    0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0...\n",
       "   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0...\n",
       "   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0...\n",
       "   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0...\n",
       "   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0...\n",
       "   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0...\n",
       "   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0...\n",
       "   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0...\n",
       "   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0...\n",
       "   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0...\n",
       "   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1...\n",
       "   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0...\n",
       "\u001b[33m...\u001b[39m\n",
       "\u001b[36mtestdata\u001b[39m: \u001b[32mSMat\u001b[39m = (    0,    0)    1\n",
       "(    1,    0)    1\n",
       "(    4,    0)    1\n",
       "(    6,    0)    1\n",
       "(    7,    0)    1\n",
       "(   11,    0)    1\n",
       "(   12,    0)    1\n",
       "(   13,    0)    1\n",
       "(   14,    0)    1\n",
       "(   18,    0)    1\n",
       "(   19,    0)    1\n",
       "(   20,    0)    1\n",
       "\u001b[33m...\u001b[39m\n",
       "\u001b[36mtestcats\u001b[39m: \u001b[32mFMat\u001b[39m =    1   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0...\n",
       "   1   0   1   0   0   0   0   0   1   0   1   0   1   0   0   0   0   0...\n",
       "   1   0   0   0   0   0   1   0   0   0   0   1   0   0   1   1   0   0...\n",
       "   1   0   0   0   0   0   1   0   0   0   0   0   0   0   0   0   0   0...\n",
       "   1   0   0   0   0   1   1   0   0   0   0   1   1   0   1   1   0   0...\n",
       "   0   1   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0...\n",
       "   0   1   1   1   1   0   0   1   0   1   1   0   0   1   0   0   1   0...\n",
       "   0   0   1   1   0   0   0   0   0   0   0   0   0   0   0   0   0   0...\n",
       "   0   0   1   1   0   0   0   0   0   0   0   0   0   0   0   0   0   0...\n",
       "   0   0   1   0   0   0   0   0   0   0   1   0   0   0   0   0   0   0...\n",
       "   0   0   1   0   0   0   0   0   0   1   1   0   0   1   0   0   0   1...\n",
       "   0   0   1   0   0   0   0   0   0   0   1   0   0   0   0   0   0   0...\n",
       "\u001b[33m...\u001b[39m\n",
       "\u001b[36mres1_5\u001b[39m: \u001b[32mSMat\u001b[39m = (    33,     0)     1\n",
       "(    47,     0)     1\n",
       "(    94,     0)     1\n",
       "(   104,     0)     1\n",
       "(   112,     0)     1\n",
       "(   118,     0)     1\n",
       "(   141,     0)     1\n",
       "(   165,     0)     1\n",
       "(   179,     0)     1\n",
       "(   251,     0)     1\n",
       "(   270,     0)     1\n",
       "(   306,     0)     1\n",
       "\u001b[33m...\u001b[39m\n",
       "\u001b[36mres1_6\u001b[39m: \u001b[32mSMat\u001b[39m = (    0,    0)    1\n",
       "(    1,    0)    1\n",
       "(    4,    0)    1\n",
       "(    6,    0)    1\n",
       "(    7,    0)    1\n",
       "(   11,    0)    1\n",
       "(   12,    0)    1\n",
       "(   13,    0)    1\n",
       "(   14,    0)    1\n",
       "(   18,    0)    1\n",
       "(   19,    0)    1\n",
       "(   20,    0)    1\n",
       "\u001b[33m...\u001b[39m"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "val dict = \"../data/rcv1/\"\n",
    "val traindata = loadSMat(dict+\"docs.smat.lz4\")\n",
    "val traincats = loadFMat(dict+\"cats.fmat.lz4\")\n",
    "val testdata = loadSMat(dict+\"testdocs.smat.lz4\")\n",
    "val testcats = loadFMat(dict+\"testcats.fmat.lz4\")\n",
    "min(traindata, 1, traindata)                       // the first \"traindata\" argument is the input, the other is output\n",
    "min(testdata, 1, testdata)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the word and document counts from the data. This turns out to be equivalent to a matrix multiply. For a data matrix A and category matrix C, we want all (cat, word) pairs (i,j) such that C(i,k) and A(j,k) are both 1 - this means that document k contains word j, and is also tagged with category i. Summing over all documents gives us\n",
    "\n",
    "$${\\rm wordcatCounts(i,j)} = \\sum_{k=1}^N C(i,k) A(j,k) = C * A^T$$\n",
    "\n",
    "\n",
    "Because we are doing independent binary classifiers for each class, we need to construct the counts for words not in the class (negwcounts).\n",
    "\n",
    "Finally, we add a smoothing count 0.5 to counts that could be zero."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[36mtruecounts\u001b[39m: \u001b[32mFMat\u001b[39m =     6601    6282       0       0    1338    2167    2931     551    3588...\n",
       "   54661   63593       0       0   35284   30807   30156    7978   40192...\n",
       "   31637   15944       0       0   18959   36288   35867   36767   18956...\n",
       "   14443   14678       0       0    7309   14194   16290    1882   11460...\n",
       "   88680   65557       0       0   54424  117815  119028   40490   68167...\n",
       "   11592   15108       0       0   14155    3868    5600    3529    2315...\n",
       "  129559  169595       0       0  164626   66692   95662  143881   53627...\n",
       "   17642   33419       0       0   39781    6437   11017   51591    7120...\n",
       "   43411   70692       0       0   69276   26538   31923   89144   15103...\n",
       "    8301    8799       0       0    5441    4137    3469    1455    2164...\n",
       "   50537  116171       0       0   53590   36965   30849   18426   31545...\n",
       "    8300    8901       0       0    5528    4220    3469    1485    2143...\n",
       "\u001b[33m...\u001b[39m\n",
       "\u001b[36mwcounts\u001b[39m: \u001b[32mFMat\u001b[39m =       6601.5      6282.5     0.50000     0.50000      1338.5      2167.5...\n",
       "       54662       63594     0.50000     0.50000       35285       30808...\n",
       "       31638       15945     0.50000     0.50000       18960       36289...\n",
       "       14444       14679     0.50000     0.50000      7309.5       14195...\n",
       "       88681       65558     0.50000     0.50000       54425  1.1782e+05...\n",
       "       11593       15109     0.50000     0.50000       14156      3868.5...\n",
       "  1.2956e+05  1.6960e+05     0.50000     0.50000  1.6463e+05       66693...\n",
       "       17643       33420     0.50000     0.50000       39782      6437.5...\n",
       "       43412       70693     0.50000     0.50000       69277       26539...\n",
       "      8301.5      8799.5     0.50000     0.50000      5441.5      4137.5...\n",
       "       50538  1.1617e+05     0.50000     0.50000       53591       36966...\n",
       "      8300.5      8901.5     0.50000     0.50000      5528.5      4220.5...\n",
       "\u001b[33m...\u001b[39m\n",
       "\u001b[36mnegwcounts\u001b[39m: \u001b[32mFMat\u001b[39m =   8.9902e+05  1.1471e+06     0.50000     0.50000  8.7687e+05  6.9242e+05...\n",
       "  8.5096e+05  1.0898e+06     0.50000     0.50000  8.4292e+05  6.6378e+05...\n",
       "  8.7398e+05  1.1375e+06     0.50000     0.50000  8.5925e+05  6.5830e+05...\n",
       "  8.9118e+05  1.1387e+06     0.50000     0.50000  8.7090e+05  6.8039e+05...\n",
       "  8.1694e+05  1.0879e+06     0.50000     0.50000  8.2378e+05  5.7677e+05...\n",
       "  8.9403e+05  1.1383e+06     0.50000     0.50000  8.6405e+05  6.9072e+05...\n",
       "  7.7606e+05  9.8381e+05     0.50000     0.50000  7.1358e+05  6.2789e+05...\n",
       "  8.8798e+05  1.1200e+06     0.50000     0.50000  8.3843e+05  6.8815e+05...\n",
       "  8.6221e+05  1.0827e+06     0.50000     0.50000  8.0893e+05  6.6805e+05...\n",
       "  8.9732e+05  1.1446e+06     0.50000     0.50000  8.7277e+05  6.9045e+05...\n",
       "  8.5508e+05  1.0372e+06     0.50000     0.50000  8.2462e+05  6.5762e+05...\n",
       "  8.9732e+05  1.1445e+06     0.50000     0.50000  8.7268e+05  6.9037e+05...\n",
       "\u001b[33m...\u001b[39m\n",
       "\u001b[36mdcounts\u001b[39m: \u001b[32mFMat\u001b[39m =     8289\n",
       "  116471\n",
       "   47402\n",
       "   25304\n",
       "  198938\n",
       "   31231\n",
       "  370541\n",
       "   79524\n",
       "  147606\n",
       "   16586\n",
       "  232297\n",
       "   16770\n",
       "\u001b[33m...\u001b[39m"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "val truecounts = traincats *^ traindata\n",
    "val wcounts = truecounts + 0.5\n",
    "val negwcounts = sum(truecounts) - truecounts + 0.5\n",
    "val dcounts = sum(traincats,2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now compute the probabilities \n",
    "* pwordcat = probability that a word is in a cat, given the cat.\n",
    "* pwordncat = probability of a word, given the complement of the cat.\n",
    "* pcat = probability that doc is in a given cat. \n",
    "* spcat = sum of pcat probabilities (> 1 because docs can be in multiple cats)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[36mpwordcat\u001b[39m: \u001b[32mFMat\u001b[39m =    0.0068321   0.0065020  5.1747e-07  5.1747e-07   0.0013853   0.0022432...\n",
       "   0.0055248   0.0064276  5.0537e-08  5.0537e-08   0.0035663   0.0031138...\n",
       "   0.0071998   0.0036285  1.1379e-07  1.1379e-07   0.0043147   0.0082583...\n",
       "   0.0065308   0.0066370  2.2608e-07  2.2608e-07   0.0033051   0.0064182...\n",
       "   0.0061666   0.0045587  3.4768e-08  3.4768e-08   0.0037845   0.0081925...\n",
       "   0.0044292   0.0057726  1.9104e-07  1.9104e-07   0.0054085   0.0014781...\n",
       "   0.0055111   0.0072142  2.1269e-08  2.1269e-08   0.0070028   0.0028369...\n",
       "   0.0050931   0.0096476  1.4434e-07  1.4434e-07    0.011484   0.0018584...\n",
       "   0.0061113   0.0099519  7.0388e-08  7.0388e-08   0.0097525   0.0037360...\n",
       "   0.0048131   0.0051018  2.8989e-07  2.8989e-07   0.0031549   0.0023989...\n",
       "   0.0019538   0.0044912  1.9330e-08  1.9330e-08   0.0020718   0.0014291...\n",
       "   0.0047402   0.0050834  2.8554e-07  2.8554e-07   0.0031572   0.0024102...\n",
       "\u001b[33m...\u001b[39m\n",
       "\u001b[36mpwordncat\u001b[39m: \u001b[32mFMat\u001b[39m =    0.0045730   0.0058350  2.5433e-09  2.5433e-09   0.0044603   0.0035221...\n",
       "   0.0045367   0.0058101  2.6657e-09  2.6657e-09   0.0044939   0.0035388...\n",
       "   0.0045245   0.0058885  2.5884e-09  2.5884e-09   0.0044482   0.0034079...\n",
       "   0.0045619   0.0058292  2.5595e-09  2.5595e-09   0.0044581   0.0034829...\n",
       "   0.0044621   0.0059417  2.7310e-09  2.7310e-09   0.0044994   0.0031503...\n",
       "   0.0045861   0.0058391  2.5648e-09  2.5648e-09   0.0044323   0.0035432...\n",
       "   0.0044596   0.0056535  2.8732e-09  2.8732e-09   0.0041006   0.0036082...\n",
       "   0.0045749   0.0057702  2.5760e-09  2.5760e-09   0.0043196   0.0035453...\n",
       "   0.0045271   0.0056848  2.6253e-09  2.6253e-09   0.0042473   0.0035076...\n",
       "   0.0045820   0.0058448  2.5532e-09  2.5532e-09   0.0044566   0.0035257...\n",
       "   0.0049805   0.0060415  2.9123e-09  2.9123e-09   0.0048031   0.0038304...\n",
       "   0.0045826   0.0058450  2.5535e-09  2.5535e-09   0.0044568   0.0035257...\n",
       "\u001b[33m...\u001b[39m\n",
       "\u001b[36mpcat\u001b[39m: \u001b[32mFMat\u001b[39m =     0.010610\n",
       "     0.14908\n",
       "    0.060673\n",
       "    0.032388\n",
       "     0.25464\n",
       "    0.039975\n",
       "     0.47428\n",
       "     0.10179\n",
       "     0.18893\n",
       "    0.021230\n",
       "     0.29733\n",
       "    0.021465\n",
       "\u001b[33m...\u001b[39m\n",
       "\u001b[36mspcat\u001b[39m: \u001b[32mFMat\u001b[39m = 3.2424"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "val pwordcat = wcounts / sum(wcounts,2)                 // Normalize the rows to sum to 1.\n",
    "val pwordncat = negwcounts / sum(negwcounts,2)          // Each row represents word probabilities conditioned on one cat. \n",
    "val pcat = dcounts / traindata.ncols\n",
    "val spcat = sum(pcat)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now take the logs of those probabilities. Here we're using the formula presented <a href=\"https://bcourses.berkeley.edu/courses/1267848/files/51512989/download?wrap=1in\">here</a> to match Naive Bayes to Logistic Regression for independent data.\n",
    "\n",
    "For each word, we compute the log of the ratio of the complementary word probability over the in-class word probability. \n",
    "\n",
    "For each category, we compute the log of the ratio of the complementary category probability over the current category probability.\n",
    "\n",
    "lpwordcat(j,i) represents $\\log\\left(\\frac{{\\rm Pr}(X_i|\\neg c_j)}{{\\rm Pr}(X_i|c_j)}\\right)$\n",
    "\n",
    "while lpcat(j) represents $\\log\\left(\\frac{{\\rm Pr}(\\neg c)}{{\\rm Pr}(c)}\\right)$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[36mlpwordcat\u001b[39m: \u001b[32mFMat\u001b[39m =    -0.40147   -0.10823    -5.3155    -5.3155     1.1693    0.45114...\n",
       "   -0.19705   -0.10100    -2.9423    -2.9423    0.23118    0.12794...\n",
       "   -0.46455    0.48418    -3.7833    -3.7833   0.030486   -0.88511...\n",
       "   -0.35877   -0.12979    -4.4811    -4.4811    0.29928   -0.61126...\n",
       "   -0.32353    0.26497    -2.5441    -2.5441    0.17303   -0.95573...\n",
       "   0.034805   0.011458    -4.3106    -4.3106   -0.19905    0.87429...\n",
       "   -0.21170   -0.24378    -2.0018    -2.0018   -0.53518    0.24048...\n",
       "   -0.10730   -0.51401    -4.0259    -4.0259   -0.97782    0.64592...\n",
       "   -0.30007   -0.55996    -3.2888    -3.2888   -0.83124  -0.063085...\n",
       "  -0.049195    0.13595    -4.7322    -4.7322    0.34544    0.38508...\n",
       "    0.93577    0.29653    -1.8927    -1.8927    0.84083    0.98593...\n",
       "  -0.033811    0.13960    -4.7169    -4.7169    0.34474    0.38036...\n",
       "\u001b[33m...\u001b[39m\n",
       "\u001b[36mlpcat\u001b[39m: \u001b[32mFMat\u001b[39m =   5.7190\n",
       "  3.0325\n",
       "  3.9597\n",
       "  4.5962\n",
       "  2.4624\n",
       "  4.3834\n",
       "  1.7641\n",
       "  3.4293\n",
       "  2.7826\n",
       "  5.0221\n",
       "  2.2930\n",
       "  5.0110\n",
       "\u001b[33m...\u001b[39m"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "val lpwordcat = ln(pwordncat/pwordcat)   // ln is log to the base e (natural log)\n",
    "val lpcat = ln((spcat-pcat)/pcat)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's where we apply Naive Bayes. The formula we're using is borrowed from <a href=\"https://bcourses.berkeley.edu/courses/1267848/files/51512989/download?wrap=1in\">here</a>.\n",
    "\n",
    "$${\\rm Pr}(c|X_1,\\ldots,X_k) = \\frac{1}{1 + \\frac{{\\rm Pr}(\\neg c)}{{\\rm Pr}(c)}\\prod_{i-1}^k\\frac{{\\rm Pr}(X_i|\\neg c)}{{\\rm Pr}(X_i|c)}}$$\n",
    "\n",
    "and we can rewrite\n",
    "\n",
    "$$\\frac{{\\rm Pr}(\\neg c)}{{\\rm Pr}(c)}\\prod_{i-1}^k\\frac{{\\rm Pr}(X_i|\\neg c)}{{\\rm Pr}(X_i|c)}$$\n",
    "\n",
    "as\n",
    "\n",
    "$$\\exp\\left(\\log\\left(\\frac{{\\rm Pr}(\\neg c)}{{\\rm Pr}(c)}\\right) + \\sum_{i=1}^k\\log\\left(\\frac{{\\rm Pr}(X_i|\\neg c)}{{\\rm Pr}(X_i|c)}\\right)\\right)  = \\exp({\\rm lpcat(j)} + {\\rm lpwordcat(j,?)} * X)$$\n",
    "\n",
    "for class number j and an input column $X$. This follows because an input column $X$ is a sparse vector with ones in the positions of the input features. The product ${\\rm lpwordcat(i,?)} * X$ picks out the features occuring in the input document and adds the corresponding logs from lpwordcat. \n",
    "\n",
    "Finally, we take the exponential above and fold it into the formula $P(c_j|X_1,\\ldots,X_k) = 1/(1+\\exp(\\cdots))$. This gives us a matrix of predictions. preds(i,j) = prediction of membership in category i for test document j. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[36mlogodds\u001b[39m: \u001b[32mFMat\u001b[39m =    -6.0932    82.044    57.881    57.881    46.718    63.467    43.108...\n",
       "    10.183    17.738    36.602    36.602    22.135    45.008    24.703...\n",
       "   -30.983    96.652    46.998    46.998    62.005    20.090   -74.427...\n",
       "   -25.657    108.47    86.850    86.850    62.880    33.790   -29.653...\n",
       "   -33.350    69.578    58.679    58.679    59.652   -65.415   -70.871...\n",
       "    78.985   -22.122    34.228    34.228    21.171    34.943    111.26...\n",
       "    36.580   -22.600   -30.684   -30.684   -2.4677    30.346    50.377...\n",
       "    92.823    85.826  -0.92882  -0.92882    35.390    113.61    124.14...\n",
       "    40.340    45.135   -29.260   -29.260    25.949    71.048    57.417...\n",
       "    48.189    14.777    32.442    32.442    32.825    70.634    87.507...\n",
       "    62.109    47.123    58.016    58.016    5.9495    89.868    117.45...\n",
       "    50.110    13.168    32.410    32.410    32.537    72.225    92.823...\n",
       "\u001b[33m...\u001b[39m\n",
       "\u001b[36mpreds\u001b[39m: \u001b[32mFMat\u001b[39m =      0.99775  2.3386e-36  7.2888e-26  7.2888e-26  5.1342e-21  2.7320e-28...\n",
       "  3.7791e-05  1.9795e-08  1.2703e-16  1.2703e-16  2.4362e-10  2.8391e-20...\n",
       "           1           0  3.8822e-21  3.8822e-21  1.1792e-27  1.8830e-09...\n",
       "           1           0  1.9122e-38  1.9122e-38  4.9173e-28  2.1134e-15...\n",
       "           1  6.0612e-31  3.2827e-26  3.2827e-26  1.2397e-26           1...\n",
       "  4.9782e-35           1  1.3651e-15  1.3651e-15  6.3936e-10  6.6751e-16...\n",
       "  1.2985e-16           1           1           1     0.92184  6.6234e-14...\n",
       "           0  5.3243e-38     0.71684     0.71684  4.2709e-16           0...\n",
       "  3.0228e-18  2.5022e-20           1           1  5.3761e-12  1.3940e-31...\n",
       "  1.1802e-21  3.8222e-07  8.1360e-15  8.1360e-15  5.5485e-15  2.1085e-31...\n",
       "  1.0628e-27  3.4240e-21  6.3677e-26  6.3677e-26   0.0026004           0...\n",
       "  1.7285e-22  1.9098e-06  8.4068e-15  8.4068e-15  7.3989e-15  4.2942e-32...\n",
       "\u001b[33m...\u001b[39m"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "val logodds = lpwordcat * testdata + lpcat\n",
    "val preds = 1 / (1 + exp(logodds))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To measure the accuracy of the predictions above, we can compute the probability that the classifier outputs the right label. We used this formula in class for the expected accuracy for logistic regression. The \"dot arrow\" operator takes dot product along rows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[36macc\u001b[39m: \u001b[32mFMat\u001b[39m =   0.96676\n",
       "  0.91988\n",
       "  0.92873\n",
       "  0.90522\n",
       "  0.92503\n",
       "  0.92083\n",
       "  0.87694\n",
       "  0.93186\n",
       "  0.91148\n",
       "  0.97116\n",
       "  0.93154\n",
       "  0.97145\n",
       "\u001b[33m...\u001b[39m\n",
       "\u001b[36mres6_1\u001b[39m: \u001b[32mFMat\u001b[39m = 0.96676,0.91988,0.92873,0.90522,0.92503,0.92083,0.87694,0.93186,0.91148,0.97116,0.93154,0.97145,0.89068,0.97366,0.96644,0.93150,0.93144,0.99561,0.96318,0.90782,0.91158,0.88406,0.97971,0.94595,0.94944,0.91045,0.90866,0.89464,0.96982,0.96171,0.93822,0.97635,0.97523,0.90473,0.92854,0.85144,0.90741,0.96143,0.98985,0.99624,0.98202,0.97888,0.99910,0.93597,0.90695,0.98017,0.96474,0.97071,0.96489,0.92164,0.99069,0.97455,0.96476,0.90273,0.97072,0.98506,0.98083,0.99524,0.97665,0.99359,0.96843,0.98735,0.97673,0.99521,0.98967,0.99410,0.93016,0.99611,0.99480,0.99684,0.99672,0.96849,0.99729,0.97603,0.96933,0.98094,0.99288,0.99335,0.99803,0.90614,0.99000,0.99077,0.99327,0.96843,0.95881,0.99586,0.99706,0.92778,0.99615,0.99851,0.99031,0.99615,0.99574,0.99295,0.99623,0.99870,0.99871,0.99938,0.99944,0.99869,0.99881,0.99952,0.99978"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "val acc = ((preds ∙→ testcats) + ((1-preds) ∙→ (1-testcats)))/preds.ncols\n",
    "acc.t"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Raw accuracy is not a good measure in most cases. When there are few positives (instances in the class vs. its complement), accuracy simply drives down false-positive rate at the expense of false-negative rate. In the worst case, the learner may always predict \"no\" and still achieve high accuracy. \n",
    "\n",
    "ROC curves and ROC Area Under the Curve (AUC) are much better. Here we compute the ROC curves from the predictions above. We need:\n",
    "* scores - the predicted quality from the formula above.\n",
    "* good - 1 for positive instances, 0 for negative instances.\n",
    "* bad - complement of good. \n",
    "* npoints (100) - specifies the number of X-axis points for the ROC plot. \n",
    "\n",
    "itest specifies which of the categories to plot for. We chose itest=6 because that category has one of the highest positive rates, and gives the most stable accuracy plots.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[36mitest\u001b[39m: \u001b[32mInt\u001b[39m = \u001b[32m6\u001b[39m\n",
       "\u001b[36mscores\u001b[39m: \u001b[32mFMat\u001b[39m = 1.2985e-16,1,1,1,0.92184,6.6234e-14,1.3227e-22,1.0000,6.4175e-05,0.74290,0.091579,2.8002e-11,4.0427e-22,0.29289,0.99790,0.0054418,0.99997,2.7469e-31,1.1725e-10,0.015976,0.95969,7.5308e-10,2.9457e-16,0.011309,9.1842e-10,2.0738e-05,3.0239e-30,3.6126e-07,2.0369e-06,1,1,0.99999,0.99999,9.8394e-11,0.81547,0.99826,5.4711e-12,0.99052,4.8879e-10,3.0592e-06,0.99899,1.0000,2.0468e-06,0.99994,0.99781,0.97165,0.0045558,0.021813,1.1182e-17,1.1182e-17,7.6746e-17,1.0000,7.6746e-17,7.3089e-06,7.3089e-06,0.99864,2.8267e-09,1.2181e-14,1,4.7808e-24,2.3243e-24,9.4091e-14,0.99724,7.0307e-08,0.99816,1.3932e-13,1.2951e-08,0.91334,6.3677e-06,5.1386e-10,3.3876e-17,1.0000,1.0000,0.99958,7.8813e-06,1.4805e-12,0.99981,0.99971,8.4397e-11,0.99939,0.99836,1,7.2057e-08,0.010264,1.0000,0.044583,8.8276e-08,6.1791e-13,0.017260,0.99966,0.0021901,4.3284e-17,0.99996,0.99821,0.93932,0.98340,4.8482e-24,1.3503e-11,2.3761e-06,0.99011,0.00018747,1,0.99882,0.98596,0.99655,0.0012145,1.5212e\u001b[33m...\u001b[39m\n",
       "\u001b[36mgood\u001b[39m: \u001b[32mFMat\u001b[39m = 0,1,1,1,1,0,0,1,0,1,1,0,0,1,0,0,1,0,1,1,1,0,0,0,0,1,0,1,0,1,1,1,1,0,1,1,0,1,0,0,1,1,1,1,1,1,1,1,0,0,0,1,0,0,0,0,1,1,1,0,0,1,1,0,1,0,0,1,1,1,0,1,1,1,1,0,1,1,0,1,1,1,0,1,1,1,0,0,1,1,0,0,1,1,1,1,0,0,0,1,1,1,1,1,1,0,0,1,0,1,1,0,0,0,0,1,1,1,1,0,1,1,1,0,0,1,1,0,1,0,1,1,1,1,0,0,0,1,0,0,1,1,1,0,0,0,0,1,1,1,1,1,1,1,1,0,0,1,0,0,1,1,1,1,1,1,1,1,1,0,1,1,1,0,0,0,1,1,1,1,1,1,0,1,0,0,1,0,1,1,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,0,0,0,0,1,1,1,1,0,0,1,0,1,1,0,1,1,1,0,0,1,1,1,1,1,1,0,1,1,0,0,1,1,1,1,1,1,0,0,1,1,0,1,1,0,0,1,0,1,1,1,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,0,1,0,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,0,1,1,0,1,0,1,0,1,1,0,1,1,1,1,1,0,1,0,0,1,1,1,1,0,1,1,1,1,0,0,0,0,0,0,0,1,1,1,1,1,0,1,0,0,1,1,0,0,1,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,0,0,1,0,1,1,1,1,1,1,1,0,1,1,1,1,1,1,1,1,0,1,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,0,1,1,1,1,1,0,1,0,1,1,1,1,0,1,1,1,1,1,0,0,0,1,0,1,1,0,1,0,1,1,1,1,1,0,1,1,\u001b[33m...\u001b[39m\n",
       "\u001b[36mbad\u001b[39m: \u001b[32mFMat\u001b[39m = 1,0,0,0,0,1,1,0,1,0,0,1,1,0,1,1,0,1,0,0,0,1,1,1,1,0,1,0,1,0,0,0,0,1,0,0,1,0,1,1,0,0,0,0,0,0,0,0,1,1,1,0,1,1,1,1,0,0,0,1,1,0,0,1,0,1,1,0,0,0,1,0,0,0,0,1,0,0,1,0,0,0,1,0,0,0,1,1,0,0,1,1,0,0,0,0,1,1,1,0,0,0,0,0,0,1,1,0,1,0,0,1,1,1,1,0,0,0,0,1,0,0,0,1,1,0,0,1,0,1,0,0,0,0,1,1,1,0,1,1,0,0,0,1,1,1,1,0,0,0,0,0,0,0,0,1,1,0,1,1,0,0,0,0,0,0,0,0,0,1,0,0,0,1,1,1,0,0,0,0,0,0,1,0,1,1,0,1,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,0,0,0,0,1,1,0,1,0,0,1,0,0,0,1,1,0,0,0,0,0,0,1,0,0,1,1,0,0,0,0,0,0,1,1,0,0,1,0,0,1,1,0,1,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,0,0,1,0,1,1,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,0,0,0,0,0,0,1,0,0,1,0,1,0,1,0,0,1,0,0,0,0,0,1,0,1,1,0,0,0,0,1,0,0,0,0,1,1,1,1,1,1,1,0,0,0,0,0,1,0,1,1,0,0,1,1,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,0,1,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,1,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,1,0,1,0,0,0,0,1,0,0,0,0,0,1,1,1,0,1,0,0,1,0,1,0,0,0,0,0,1,0,0,\u001b[33m...\u001b[39m\n",
       "\u001b[36mrr\u001b[39m: \u001b[32mDMat\u001b[39m =         0\n",
       "  0.72112\n",
       "  0.77285\n",
       "  0.79854\n",
       "  0.81708\n",
       "  0.83275\n",
       "  0.84081\n",
       "  0.84990\n",
       "  0.85898\n",
       "  0.86510\n",
       "  0.87159\n",
       "  0.87762\n",
       "\u001b[33m...\u001b[39m"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "val itest = 6\n",
    "val scores = preds(itest,?)\n",
    "val good = testcats(itest,?)\n",
    "val bad = 1-testcats(itest,?)\n",
    "val rr =roc(scores,good,bad,100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> TODO 1: In the cell below, write an expression to derive the ROC Area under the curve (AUC) given the curve rr. rr gives the ROC curve y-coordinates at 100 evenly-spaced X-values from 0 to 1.0. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "// auc = "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> TODO 2: In the cell below, write the value of AUC returned by the expression above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Logistic Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now lets train a logistic classifier on the same data. BIDMach has an umbrella classifier called GLM for Generalized Linear Model. GLM includes linear regression, logistic regression (with log accuracy or direct accuracy optimization), and SVM. \n",
    "\n",
    "The learner function accepts these arguments:\n",
    "* traindata: the training data in the same format as for Naive Bayes\n",
    "* traincats: the training category labels\n",
    "* testdata: the test input data\n",
    "* predcats: a container for the predictions generated by the model\n",
    "* modeltype (GLM.logistic here): an integer that specifies the type of model (0=linear, 1=logistic log accuracy, 2=logistic accuracy, 3=SVM). \n",
    "\n",
    "We'll construct the learner and then look at its options:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Option Name       Type          Value\n",
      "===========       ====          =====\n",
      "addConstFeat      boolean       false\n",
      "aopts             Opts          null\n",
      "autoReset         boolean       true\n",
      "batchSize         int           10000\n",
      "checkPointFile    String        null\n",
      "checkPointInterval  float         0.0\n",
      "clipByValue       float         -1.0\n",
      "cumScore          int           0\n",
      "debug             int           0\n",
      "debugCPUmem       boolean       false\n",
      "debugMem          boolean       false\n",
      "dim               int           256\n",
      "doAllReduce       boolean       false\n",
      "doubleScore       boolean       false\n",
      "doVariance        boolean       false\n",
      "epsilon           float         1.0E-5\n",
      "evalStep          int           11\n",
      "featThreshold     Mat           null\n",
      "featType          int           1\n",
      "gsq_decay         float         -1.0\n",
      "hashBound1        int           1000000\n",
      "hashBound2        int           1000000\n",
      "hashFeatures      int           0\n",
      "initsumsq         float         1.0E-5\n",
      "iweight           FMat          null\n",
      "l2reg             FMat          null\n",
      "langevin          float         0.0\n",
      "lim               float         0.0\n",
      "links             IMat          2,2,2,2,2,2,2,2,2,2,...\n",
      "logDataSink       DataSink      null\n",
      "logfile           String        log.txt\n",
      "logFuncs          Function2[]   null\n",
      "lr_policy         Function3     null\n",
      "lrate             FMat          1\n",
      "mask              FMat          null\n",
      "matrixOfScores    boolean       false\n",
      "max_grad_norm     float         -1.0\n",
      "mixinInterval     int           1\n",
      "naturalLambda     float         0.0\n",
      "nesterov_vel_decay  FMat          null\n",
      "nNatural          int           1\n",
      "npasses           int           2\n",
      "nzPerColumn       int           0\n",
      "pauseAt           long          -1\n",
      "pexp              FMat          0\n",
      "pstep             float         0.01\n",
      "putBack           int           -1\n",
      "r1nmats           int           1\n",
      "reg1weight        FMat          1.0000e-07\n",
      "resFile           String        null\n",
      "rmask             FMat          null\n",
      "sample            float         1.0\n",
      "sizeMargin        float         3.0\n",
      "startBlock        int           8000\n",
      "targets           FMat          null\n",
      "targmap           FMat          null\n",
      "texp              FMat          0.50000\n",
      "trace             int           0\n",
      "updateAll         boolean       false\n",
      "useCache          boolean       true\n",
      "useDouble         boolean       false\n",
      "useGPU            boolean       true\n",
      "useGPUcache       boolean       true\n",
      "vel_decay         FMat          null\n",
      "vexp              FMat          0.50000\n",
      "waitsteps         int           3\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "\u001b[36mpredcats\u001b[39m: \u001b[32mFMat\u001b[39m =    0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0...\n",
       "   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0...\n",
       "   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0...\n",
       "   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0...\n",
       "   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0...\n",
       "   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0...\n",
       "   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0...\n",
       "   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0...\n",
       "   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0...\n",
       "   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0...\n",
       "   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0...\n",
       "   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0...\n",
       "\u001b[33m...\u001b[39m\n",
       "\u001b[36mmm\u001b[39m: \u001b[32mLearner\u001b[39m = BIDMach.Learner@6b2a7def\n",
       "\u001b[36mmopts\u001b[39m: \u001b[32mGLM\u001b[39m.\u001b[32mLearnOptions\u001b[39m = BIDMach.models.GLM$LearnOptions@6032d02c"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "val predcats = zeros(testcats.nrows, testcats.ncols)\n",
    "val (mm,mopts) = GLM.learner(traindata, traincats, GLM.maxp)\n",
    "mopts.what"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The most important options are:\n",
    "* lrate: the learning rate\n",
    "* batchSize: the minibatch size\n",
    "* npasses: the number of passes over the dataset\n",
    "\n",
    "We'll use the following parameters for this training run. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "corpus perplexity=81528.088805\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "11:04:55 INFO: pass= 0\n",
      "11:04:57 INFO:  1.00%, score=-0.50000, secs=2.6, samps/s=3117.7, gf= 0.9, MB/s=3.34, GPUmem=0.352330\n",
      "11:04:59 INFO:  2.00%, score=-0.02578, secs=4.3, samps/s=3702.8, gf= 1.2, MB/s=3.91, GPUmem=0.352330\n",
      "11:05:01 INFO:  3.00%, score=-0.02423, secs=6.0, samps/s=4014.0, gf= 1.3, MB/s=4.22, GPUmem=0.352330\n",
      "11:05:03 INFO:  4.00%, score=-0.02123, secs=8.3, samps/s=4209.3, gf= 1.4, MB/s=4.42, GPUmem=0.352330\n",
      "11:05:05 INFO:  5.00%, score=-0.02148, secs=10.7, samps/s=4317.6, gf= 1.4, MB/s=4.54, GPUmem=0.352330\n",
      "11:05:08 INFO:  7.00%, score=-0.02072, secs=13.0, samps/s=4387.7, gf= 1.5, MB/s=4.61, GPUmem=0.352330\n",
      "11:05:10 INFO:  8.00%, score=-0.02275, secs=15.3, samps/s=4437.2, gf= 1.5, MB/s=4.66, GPUmem=0.352330\n",
      "11:05:12 INFO: 10.00%, score=-0.02079, secs=17.7, samps/s=4472.4, gf= 1.5, MB/s=4.70, GPUmem=0.352330\n",
      "11:05:15 INFO: 11.00%, score=-0.02122, secs=20.0, samps/s=4500.7, gf= 1.5, MB/s=4.73, GPUmem=0.352330\n",
      "11:05:17 INFO: 12.00%, score=-0.02124, secs=22.3, samps/s=4523.5, gf= 1.5, MB/s=4.73, GPUmem=0.352330\n",
      "11:05:19 INFO: 14.00%, score=-0.01956, secs=24.7, samps/s=4542.1, gf= 1.5, MB/s=4.73, GPUmem=0.352330\n",
      "11:05:22 INFO: 15.00%, score=-0.01604, secs=27.0, samps/s=4556.7, gf= 1.6, MB/s=4.73, GPUmem=0.352330\n",
      "11:05:24 INFO: 17.00%, score=-0.01926, secs=29.3, samps/s=4569.9, gf= 1.6, MB/s=4.73, GPUmem=0.352330\n",
      "11:05:26 INFO: 18.00%, score=-0.01616, secs=31.7, samps/s=4580.2, gf= 1.6, MB/s=4.74, GPUmem=0.352330\n",
      "11:05:29 INFO: 19.00%, score=-0.01942, secs=34.0, samps/s=4589.0, gf= 1.6, MB/s=4.74, GPUmem=0.352330\n",
      "11:05:31 INFO: 21.00%, score=-0.01935, secs=36.3, samps/s=4597.5, gf= 1.6, MB/s=4.73, GPUmem=0.352330\n",
      "11:05:33 INFO: 22.00%, score=-0.01776, secs=38.7, samps/s=4604.4, gf= 1.6, MB/s=4.73, GPUmem=0.352330\n",
      "11:05:36 INFO: 24.00%, score=-0.01736, secs=41.3, samps/s=4581.2, gf= 1.6, MB/s=4.70, GPUmem=0.352330\n",
      "11:05:38 INFO: 25.00%, score=-0.01787, secs=43.6, samps/s=4588.3, gf= 1.6, MB/s=4.72, GPUmem=0.352330\n",
      "11:05:41 INFO: 27.00%, score=-0.02167, secs=45.9, samps/s=4594.7, gf= 1.6, MB/s=4.72, GPUmem=0.352330\n",
      "11:05:43 INFO: 28.00%, score=-0.01949, secs=48.3, samps/s=4600.1, gf= 1.6, MB/s=4.73, GPUmem=0.352330\n",
      "11:05:45 INFO: 29.00%, score=-0.01993, secs=50.6, samps/s=4604.8, gf= 1.6, MB/s=4.74, GPUmem=0.352330\n",
      "11:05:48 INFO: 31.00%, score=-0.01999, secs=53.0, samps/s=4600.1, gf= 1.6, MB/s=4.73, GPUmem=0.352330\n",
      "11:05:50 INFO: 32.00%, score=-0.01648, secs=55.5, samps/s=4594.1, gf= 1.6, MB/s=4.73, GPUmem=0.352330\n",
      "11:05:52 INFO: 34.00%, score=-0.01744, secs=57.8, samps/s=4598.7, gf= 1.6, MB/s=4.74, GPUmem=0.352330\n",
      "11:05:55 INFO: 35.00%, score=-0.01929, secs=60.2, samps/s=4602.6, gf= 1.6, MB/s=4.75, GPUmem=0.352330\n",
      "11:05:57 INFO: 36.00%, score=-0.01943, secs=62.5, samps/s=4606.6, gf= 1.6, MB/s=4.75, GPUmem=0.352330\n",
      "11:05:59 INFO: 38.00%, score=-0.01960, secs=64.9, samps/s=4610.4, gf= 1.6, MB/s=4.76, GPUmem=0.352330\n",
      "11:06:02 INFO: 39.00%, score=-0.01815, secs=67.2, samps/s=4614.1, gf= 1.6, MB/s=4.76, GPUmem=0.352330\n",
      "11:06:04 INFO: 41.00%, score=-0.02070, secs=69.5, samps/s=4617.6, gf= 1.6, MB/s=4.76, GPUmem=0.352330\n",
      "11:06:06 INFO: 42.00%, score=-0.01609, secs=71.8, samps/s=4620.7, gf= 1.6, MB/s=4.76, GPUmem=0.352330\n",
      "11:06:09 INFO: 43.00%, score=-0.01814, secs=74.2, samps/s=4623.5, gf= 1.6, MB/s=4.77, GPUmem=0.352330\n",
      "11:06:11 INFO: 45.00%, score=-0.01705, secs=76.5, samps/s=4626.3, gf= 1.6, MB/s=4.77, GPUmem=0.352330\n"
     ]
    }
   ],
   "source": [
    "mopts.lrate=1.0\n",
    "mopts.batchSize=1000\n",
    "mopts.npasses=2\n",
    "mm.train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "val (nn, nopts) = GLM.predictor(mm.model, testdata)\n",
    "\n",
    "nn.predict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "val predcats = FMat(nn.preds(0))\n",
    "val lacc = (predcats ∙→ testcats + (1-predcats) ∙→ (1-testcats))/preds.ncols\n",
    "lacc.t\n",
    "mean(lacc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we have the accuracy scores for both Naive Bayes and Logistic regression, we can plot both of them on the same axes. Naive Bayes is red, Logistic regression is blue. The x-axis is the category number from 0 to 102. The y-axis is the absolute accuracy of the predictor for that category. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "val axaxis = row(0 until 103)\n",
    "plot(axaxis, acc, axaxis, lacc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> TODO 3: With the full training set (700k training documents), Logistic Regression is noticeably more accurate than Naive Bayes in every category. What do you observe in the plot above? Why do you think this is?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we'll compute the ROC plot and ROC area (AUC) for Logistic regression  for category itest."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "val lscores = predcats(itest,?)\n",
    "val lrr =roc(lscores,good,bad,100)\n",
    "val auc = mean(lrr)                           // Fill in using the formula you used before"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We computed the ROC curve for Naive Bayes earlier, so now we can plot them on the same axes. Naive Bayes is once again in red, Logistic regression in blue. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "val rocxaxis = row(0 until 101)\n",
    "plot(rocxaxis, rr, rocxaxis, lrr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">TODO 4: In the cell below, compute and plot lift curves from the ROC curves for Naive Bayes and Logistic regression. The lift curves should show the ratio of ROC y-values over a unit slope diagonal line (Y=X). The X-values should be the same as for the ROC plots, except that X=0 will be omitted since the lift will be undefined. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> TODO 5: Experiment with different values for learning rate and batchSize to get the best performance for absolute accuracy and ROC area on category 6. Write your optimal values below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "BIDMach",
   "language": "bidmach",
   "name": "bidmach"
  },
  "language_info": {
   "codemirror_mode": "text/x-scala",
   "file_extension": ".scala",
   "mimetype": "text/x-scala",
   "name": "scala211",
   "nbconvert_exporter": "script",
   "pygments_lexer": "scala",
   "version": "2.11.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
